Differentially Private Ordinary Least Squares: $t$-Values, Confidence Intervals and Rejecting Null-Hypotheses

Linear regression is one of the most prevalent techniques in data analysis. Given a large collection of samples composed of features x and a label y, linear regression is used to find the best prediction of the label as a linear combination of the features. However, it is also common to use linear regression for its explanatory capabilities rather than label prediction. Ordinary Least Squares (OLS) is often used in statistics to establish a correlation between an attribute (e.g. gender) and a label (e.g. income) in the presence of other (potentially correlated) features. OLS uses linear regression in order to estimate the correlation between the label and a feature xj on a given dataset. Then, under the assumption of a certain random generative model for the data, OLS derives t-values — representing the likelihood of each real value to be the true correlation in the underlying distribution. Using t-values, OLS can release a confidence interval, which is an interval on the reals that is likely to contain the true correlation. When this interval does not intersect the origin, we can reject the null hypothesis as it is likely that xj indeed has a non-zero correlation with y. Our work aims at achieving similar guarantees on data under differentially private estimators. We use the Gaussian Johnson-Lindenstrauss transform, which has been shown to satisfy differential privacy if the given data has large singular values [BBDS12]. We analyze the result of projecting the data using the JL transform under the OLS model and show how to derive approximated t-values. Using the approximated t-values we give confidence intervals and bound the number of samples needed to reject the null hypothesis with differential privacy, when the data is drawn i.i.d from a multivariate Gaussian. When not all singular values of the data are sufficiently large, we alter the input and increase its singular values and then project it using a JL transform. Thus our projected data yields an approximation for the Ridge Regression problem — a variant of the linear regression that uses a l2-regularization term. We derive, under certain conditions, confidence intervals using the projected Ridge regression. We also derive, under different conditions, confidence intervals for the “Analyze Gauss” algorithm of Dwork et al [DTTZ14].